Affect and Emotions in Political Trust

Today’s Agenda

1. Audio Data in the Social Sciences and in Surveys: What insights can we gain by leveraging spoken (audio) responses in surveys?
1. Emotion Analysis: What insights can we gain through automated emotion analysis of audio responses?
1. Paper: Affect and Emotions in Political Trust

Audio Data in the Social Sciences and in Surveys

What is Audio Data?

Audio data = information represented in the form of sound waves captured in a digital format (e.g., WAV)

Figure 1: Waveform of an audio file (amplitude over time).

Sources for audio data for the social sciences: Interviews, social media (e.g. live streams, Youtube, TikTok), public speeches and debates, political talk shows and podcasts, press conferences, recorded survey answers.

What are characteristics of Audio Data?

Audio Data contains rich information including paralinguistic elements
- e.g., pitch, volume, laughter and sighs, tone of voice, pause and silence, emotional cues

“System 1” versus “System 2” language/answers (Lütters et al. 2018)
- System 1: intuitive, automatic, and fast thinking → spoken answers
- System 2: analytical, deliberate, and slow thinking → written answers

Audio Answers in Surveys

Spoken answers compared to written answers are longer, more elaborate and detailed (Gavras et al. 2022, Höhne & Gavras 2022, Lütters et al. 2018, Revilla et al. 2020)

Explanations:

Gavras et al. 2022: Audio formats facilitate the answer process by enabling open narration, intuitive and spontaneous answers (≠ intentional and conscious text answers).
Revilla et al. 2020: Speaking requires less effort than typing and voice formats make survey answering easier/quicker (barriers!).

Unfortunately, spoken answers compared to written answers increase response times and non-response rates (Revilla et al. 2020, Lütters et al. 2018)

Methods for the Analysis of Audio Data

Increasing number of methods to analyze audio data (Computer Science, Computational Linguistics, Computational Social Science)

Examples of Methods:
- Natural Language Processing
- Automatic Speech Recognition (“speech-to-text”) (Landesvatter et al. 2023)
- Speaker Diarization / Speaker Identification
- Environmental Sound Analysis
- Speech Emotion Recognition

Emotion Analysis

Sentiment Analysis = Emotion Recognition?

Affect = “an umbrella term that is used to refer to both emotions and moods” (Lee, Dirks, and Campagna 2023, 549)
Sentiment: the valence of a feeling (e.g., positive versus negative)

The concept of emotions is not always clearly distinguished from similar phenomena such as mood, affect, and feeling. (Gabriel et al. 2023, 39)

Emotions: more complex and multi-dimensional state of feeling characterized by their intensity driven by different cognitive evaluations (e.g., anger towards politicians)

Speech Emotion Recognition

Why use speech (audio data) for Emotion Analysis?
- → Recognizing emotions from text is difficult because it is stripped of all paralinguistic and acoustic features

Speech Emotion Recognition (SER) is part of the field of Automated Emotion Recognition (intersect with disciplines such as Neurocomputing, Affective Computing)

Evolution of Models from traditional Machine Learning models (e.g., logistic regression) to methods based on deep learning/neural networks, e.g. SpeechBrain (Ravanelli et al. 2021)

Paper: Affect and Emotions in Political Trust

Research Question and Motivation (1)

Research question: Do sentiment & emotions impact political trust judgments?

Substantive debate: Rational reasoning vs. affective states
- Lahno (2020): Game-theory → rational choice assumptions & risk calculations & social trust
- “affect-based” form of political trust (e.g., Theiss-Morse and Barton 2017)

Research Question and Motivation (2)

Today’s relevance of emotions
- Changing media environment → emotions (fear, anger/rage) used to influence people (and their trust), e.g., by providing emotionally arousing content such as immgrants’ crime (Gabriel et al. 2023)

Methodological:
- Extend regression-based analysis through open-ended probing data
- Explore audio (and text) data to measure emotions & their influence on political trust

Theory and hypotheses

“A decision to trust a government organization may […] not always be conscious and/or rational.” (Grimmelikhuijsen 2012, 57)
“Automatic Hot Cognition” (Lodge and Taber 2013): all sociopolitical concepts that have been evaluated in the past are affectively charged → affective charge is automatically activated within milliseconds on mere exposure to the concept
H1: Open-ended survey answers to the question about the reasons for respondents’ trust include both non-neutral sentiment (H1a) as well as emotional, i.e., non-neutral language (H1b)
H2: The sentiment or emotion expressed in an open-ended survey answer is correlated with the closed-ended trust score
- Positive (negative) sentiment has a positive (negative) effect on political trust (H2a)
- Emotion of happiness (anger/sadness) has a positive (negative) effect on political trust (H2b)

Questionnaire Design

Political Trust Question	“How often can you trust the federal government in Washington to do what is right?” (ANES)	5 closed-ended categories [Never (0); Some of the time (1); About half of the time (2); Most of the time (3); Always (4)]
Probing Question	“The previous question was: ‘How often can you trust the federal government in Washington to do what is right?’. Your answer was: ‘About half of the time’. In your own words, please explain why you selected this answer.”	open-ended, audio request, SVoice tool (Höhne, Gavras and Qureshi 2021)

Data: Overview

Self-administered web survey (prolific), September 6 to October 27, 2023
U.S. non-probability sample; \(n\)=1,474 with 491 open-ended audio answers (33% response rate)
quota-based (U.S. Census Bureau 2015) with challenges in obtaining sufficient participants in the oldest age category (58+)

Methods

Data: Example

Political trust (0-4)	Probing question	Emotion (Speechbrain)	Sentiment (BERT)	Sentiment (GPT)
0	Government exists only to perpetuate itself. It does so by promulgating rules and raising taxes in order to keep control of the populace.	Anger	negative	negative
1	It is very hard for the left and the right to come together to make decisions that affect our lives. That is why I only trust the government some of the time.	Anger	negative	neutral
3	Because we are the strongest in the world. And we have ever done things that people's America has like.	Happiness	positive	positive
0	The federal government, like all the other governments on this planet, is corrupt and it will never have your best interest in mind. Politicians are corrupt evil people.	Neutral	negative	negative
3	Again, Congress does what's best for the public. It's only fringe issues that make the news. Most of the stuff goes to them. Most people don't pay attention to it.	Neutral	neutral	positive
1	I don't have too much trust in the government these days. I think there's too much money involved in influencing politicians and laws. So I should have said never, but sometimes maybe.	Sadness	negative	negative
3	I think if you take the craziness and the partisan politics out of the entire thing, there's an awful lot of people that are good people that work in the different areas of federal government. I think they have the best interest at heart of most of the people in the country.	Sadness	positive	positive

Results: Sentiment distribution (H1a, text data)

The commonly used trust in government question used in many surveys produces predominantly negative associations.

Figure 2: Sentiment Classification with three categories by classifier (BERT vs. GPT).
*Note.* n=491 open-ended answers.

Results: Sentiment & political trust (H2a, text data)

Associations with a positive sentiment positively influence trust scores, and vice versa
- → Associations with the question wording and thought processes matter.

Figure 3: Linear model of sentiment and a five-category trust score (bi- and multivariate).
*Note.* GPT-based classification. Reference category is negative sentimemt.

Results: Emotions distribution (H1b, audio data)

Only 17% of the probing answers contain emotional language.

Figure 4: Emotion Classification obtained from SpeechBrain.
*Note.* Analysis of n=491 open-ended answers. Number of observations for each sentiment category: 408 (neutral), 44 (angry), 18 (sad), 21 (happy).

Results: Emotions & political trust (H2b, audio data)

Respondents with a happy emotional condition have increased trust scores.

Figure 5: Linear model of emotion and a five-category trust score (bi- and multivariate).
*Note.* SpeechBrain-based classification.

Conclusion and discussion

Sentiment is related to trust judgments, but no consistent effect of emotions (only happiness)
NLP/AI introduces new opportunities for measurement

Limitations and follow-up questions:

Other probing wordings; Small response rates (33%); Discrete emotions; Combine text & audio/sound;

Survey setting not emotion enough? What is the setting?

How does SpeechBrain classify? Can we generate a human benchmark?

How could future research on voice and emotions look like?

Outlook on future research agenda

Audio data plays a crucial role for studying societies, as spoken language is one of humanity’s most important means of communication, expression and information exchange in various fields (e.g., public speeches and debates, political talk shows and podcasts, press conferences).
→ We need more applied research to gain more experiences with this type of data.

Subfields to research: speech-to-text algorithms (Landesvatter, Behnert, Bauer 2023, Meitinger, Sluis, Schonlau 2024), ethical considerations and privacy concerns, survey and experiments design to collect audio data, etc.

Multi-disciplinary approaches are useful (e.g., experts from linguistics, sociology, psychology, and computer science).

Speech-to-Text algorithms

Landesvatter, Camille, Jan Behnert, and Paul C. Bauer. 2023. “Comparing Speech-to-text Algorithms for Transcribing Voice Data from Surveys.” SocArXiv. October 10. doi:10.31235/osf.io/vk6wj.

Thank you for your Attention!

References

Gabriel, Maier, Masch, and Renner. 2023. Political Leaders, the Display of Emotions, and the Public: An Empirical Study on the Coverage and Effects of Emotions in German Politics. Nomos.

Gavras, Höhne, Blom, and Schoen. 2022. “Innovating the collection of open-ended answers: The linguistic and content characteristics of written and oral answers to political attitude questions.” Journal of the Royal Statistical Society. Series A, 185(3):872-890.

Grimmelikhuijsen, Stephan. 2012. “Linking Transparency, Knowledge and Citizen Trust in Government: An Experiment.” International Review of Administrative Sciences 78(1):50–73.

Höhne and Gavras. 2022. “Typing or Speaking? Comparing Text and Voice Answers to Open Questions on Sensitive Topics in Smartphone Surveys.” Available at SSRN: https://ssrn.com/abstract=4239015 or http://dx.doi.org/10.2139/ssrn.4239015.

Lee, Kurt, and Rachel L. Campagna. 2023. “At the Heart of Trust: Understanding the Integral Relationship Between Emotion and Trust.” Group & Organization Management 48(2):546–80.

Lodge, Milton, and Charles S. Taber. 2013. The Rationalizing Voter. Cambridge University Press.

Lütters, Friedrich-Freksa, and Egger. 2018.“Effects of Speech Assistance in Online Questionnaires.” Presented at the General Online Research Conference, Vol. 18.

Ravanelli, Parcollet, Plantinga, et al. 2021. “SpeechBrain: A General-Purpose Speech Toolkit.” arXiv.

Revilla, Couper, Bosch, and Asensio. 2020. “Testing the Use of Voice Input in a Smartphone Web Survey.” Social Science Computer Review 38(2):207–24.

Theiss-Morse and Dona-Gene Barton. 2017. “Emotion, Cognition, and Political Trust.” Pp. 160–75 in Handbook on Political Trust. Edward Elgar Publishing.